Using Expert Assessments to Estimate Probability Distributions

Previous decision analysis literature has established the presence of judgmental errors in the quantile judgments provided by an expert. However a systematic approach to using quantile judgments to deduce the mean and standard deviation of the underlying probability distribution while accounting for judgmental errors has not yet been established. This paper develops an optimization based approach to accomplish this task. The approach estimates both the mean and standard deviation as weighted linear combinations of quantile judgments, where the weights are explicit functions of the expert’s judgmental errors. The approach is analytically tractable, and provides flexibility to elicit any set of quantiles from an expert. The structural properties of the solution explain a number of numerical observations made in the extant decision analysis literature. The approach also establishes that using an expert’s quantile judgments to deduce the distribution parameters is equivalent to collecting data with a specific sample size. The approach allows combining the expert’s judgments with data in a Bayesian fashion and with other experts’ estimates in a manner that is consistent with the existing theory of combining point forecasts from noisy sources. The paper also contributes to practice by discussing a large-scale implementation of the theory at Dow AgroSciences to make an annual $800 million decision, and providing insights into the selection of quantiles to elicit from an expert. Results show that the judgments of the expert at Dow are equivalent to 5–6 years of data collection, with substantial monetary savings.